Method and system for scoring articles in an on-demand services environment

ABSTRACT

A method for scoring and ranking knowledge articles in an on-demand service environment in a database network system. The scoring allows users to have input into the scoring to determine relevancy and authenticity to solve particular problems.

CLAIM OF PRIORITY

This patent document is a continuation of and claims priority toco-pending and commonly assigned U.S. patent application Ser. No.13/082,299, titled “Method and System for Scoring Articles in anOn-Demand Services Environment,” by Rajagopal, filed on Apr. 7, 2011(Attorney Docket No. SLFCP150/436US), which claims priority to commonlyassigned U.S. Provisional Patent Application No. 61/360,752, titled“Method and System for Scoring and Ranking Articles in an On-DemandServices Environment,” by Rajagopal, filed on Jul. 1, 2010. Each of U.S.patent application Ser. No. 13/082,299 and U.S. Provisional PatentApplication No. 61/360,752 is hereby incorporated by reference in itsentirety and for all purposes.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

CROSS REFERENCE TO RELATED APPLICATIONS

The following commonly owned, co-pending United States patents andpatent applications, including the present application, are related toeach other. Each of the other patents/applications are incorporated byreference herein in its entirety:

U.S. Utility patent application Ser. No. 13/082,299 entitled METHOD ANDSYSTEM FOR SCORING ARTICLES IN AN ON-DEMAND SERVICES ENVIRONMENT, byVinodh Rajagopal, filed Apr. 7, 2011 (Docket#48-52/436US); and

U.S. Provisional Patent Application No. 61/354,003 entitled METHODS ANDSYSTEMS FOR ANALYZING SEARCH TERMS IN A MULTI-TENANT DATABASE SYSTEMENVIRONMENT, by Olivier Y. Pin et al., filed Jun. 11, 2010 (AttorneyDocket No. SALEP0012P); and

U.S. patent application Ser. No. 11/716,365 entitled METHOD AND SYSTEMFOR POSTING IDEAS, by Norimasa Yoshida et al., filed Mar. 8, 2007(Attorney Docket No. 48-2/041 US), which is now U.S. Pat. No. 7,831,455;and

U.S. patent application Ser. No. 11/786,882 entitled METHOD AND SYSTEMFOR MANAGING IDEAS, by Norimasa Yoshida et al., filed Apr. 13, 2007(Attorney Docket No. 48-3/042US), which is now U.S. Pat. No. 7,818,194;and

U.S. patent application Ser. No. 11/801,572 entitled METHOD AND SYSTEMFOR INTEGRATING IDEA AND ON DEMAND SERVICES, by Norimasa Yoshida et al.,filed May 9, 2007 (Attorney Docket No. 48-4/043US), which is now U.S.Pat. No. 7,840,413; and

U.S. Provisional Patent Application No. 61/354,604 entitled METHODS ANDSYSTEMS FOR DYNAMICALLY SUGGESTING ANSWERS TO QUESTIONS SUBMITTED TO APORTAL OF AN ONLINE SERVICE, by Patrick McFarlane et al., filed Jun. 14,2010 (Attorney Docket No. SALEP0030P); and

U.S. patent application Ser. No. 13/111,195 entitled METHODS AND SYSTEMSFOR DYNAMICALLY SUGGESTING ANSWERS TO QUESTIONS SUBMITTED TO A PORTAL OFAN ONLINE SERVICE, by Patrick McFarlane et al., filed May 19, 2011(Attorney Docket No. 48-51/429US); and

U.S. Provisional Patent Application No. 61/360,752 entitled METHODS ANDSYSTEMS FOR SCORING AND RANKING ARTICLES IN AN ON-DEMAND SERVICESENVIRONMENT, by Vinodh Rajagopal, filed Jul. 1, 2010 (Attorney DocketNo. SALEP0034P); and

U.S. patent application Ser. No. 13/082,299 entitled METHOD AND SYSTEMFOR SCORING ARTICLES IN AN ON-DEMAND SERVICES ENVIRONMENT, by VinodhRajagopal, filed Apr. 7, 2011 (Attorney Docket No. 48-52/436US); and

U.S. Provisional Patent Application No. 61/351,620 entitled METHODS ANDSYSTEMS FOR PROVIDING ANSWERS TO USERS OF A MULTI-TENANT DATABASESYSTEM, by Norimasa Yoshida et al., filed Jun. 4, 2010 (Attorney DocketNo. SALEP0004P); and

U.S. patent application Ser. No. 13/152,246 entitled METHOD AND SYSTEMFOR PROVIDING ANSWERS TO USERS OF A MULTI-TENANT DATABASE SYSTEM, byNorimasa Yoshida et al., filed Jun. 6, 2011 (Attorney Docket No.48-54/384US); and

U.S. Provisional Patent Application No. 61/350,904 entitled METHOD ANDSYSTEM FOR ANSWERS TO CASES ESCALATION, by Norimasa Yoshida et al.,filed Jun. 2, 2010 (Attorney Docket No. SALEP0005P); and

U.S. patent application Ser. No. 13/151,784 entitled METHOD AND SYSTEMFOR ESCALATING CONTENT OF DISCUSSIONS TO PARTICULAR MEMORY LOCATIONS, byNorimasa Yoshida et al., filed Jun. 6, 2011 (Attorney Docket No.48-55/385US).

FIELD OF THE INVENTION

The current invention relates generally to scoring and ranking articles.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches, which in and of themselves may also be inventions.

In conventional database systems, users access their data resources inone logical database. A user of such a conventional system typicallyretrieves data from and stores data on the system using the user's ownsystems. A user system might remotely access one of a plurality ofserver systems that might in turn access the database system. Dataretrieval from the system might include the issuance of a query from theuser system to the database system. The database system might processthe request for information received in the query and send to the usersystem information relevant to the request. The easy and efficientretrieval of accurate information and subsequent delivery of thisinformation to the user system has been and continues to be a goal ofadministrators of database systems.

Unfortunately, conventional database approaches might not process aquery as efficiently as the user might want if, for example, the resultsof the query are not organized correctly. Accordingly, it is desirableto provide techniques enabling better and more relevant organization ofthe results of a query.

BRIEF SUMMARY

In accordance with embodiments, there are provided mechanisms andmethods for scoring and ranking knowledge articles in an on-demanddatabase. These mechanisms and methods for scoring and ranking knowledgearticles in an on-demand database can enable embodiments to provideinformation in response to a user's search query in a manner that allowsinput from internal and external users. The ability of embodiments toprovide user with relevant results as a result of the scoring andranking of the knowledge articles provides an advantage over othersystems.

In an embodiment and by way of example, a method for scoring and rankingknowledge articles in an on-demand database is provided. The methodembodiment includes methods for scoring and/or ranking knowledgearticles in a multi-tenant database system, by a combination ofdetermining a weight based on voting for each of a plurality ofknowledge articles, determining a score based on reference to otherknowledge articles for each knowledge article; determining a weightbased on visitation by external users for each knowledge article;determining a half-life for each of the knowledge articles; and/ordetermining a half-life to obtain a final score. In some embodiments,the method also includes receiving a search request from a user of theon-demand services environment; ranking the knowledge articles based onthe final score and the relevancy to the search; and providing one ormore of the knowledge articles to the user based on the ranking.

Alternative method embodiments include methods for determining, by ahost system of a multitenant database having a processor systemincluding one or more processors and a storage system, an increment fora score of an article based on receiving at least one vote for thearticle; determining, by the processor system, an increment for a scoreof an article based on a creation of a reference to the article in oneor more other articles; determining, by the host system, an incrementfor a score of an article based on a detection of one or morevisitations by one or more users; weighting, by the host system, theincrement to the score, so that more recent increments to the score havea higher weight than less recent increments; and storing the score inassociation with the article in the storage system as an indication of alikelihood that users will be interested in the article.

While the present invention is described with reference to an embodimentin which techniques for scoring and ranking knowledge articles in anon-demand database are implemented in a system having an applicationserver providing a front end for an on-demand database service capableof supporting multiple tenants, the present invention is not limited tomulti-tenant databases nor deployment on application servers. In anembodiment the tenants may be organizations with employees andcustomers, whom may be users of the multitenant database as a result ofthe organization being a tenant of the multitenant database. Embodimentsmay be practiced using other database architectures, i.e., ORACLE®, DB2®by IBM and the like without departing from the scope of the embodimentsclaimed.

Any of the above embodiments may be used alone or together with oneanother in any combination. Inventions encompassed within thisspecification may also include embodiments that are only partiallymentioned or alluded to or are not mentioned or alluded to at all inthis brief summary or in the abstract. Although various embodiments ofthe invention may have been motivated by various deficiencies with theprior art, which may be discussed or alluded to in one or more places inthe specification, the embodiments of the invention do not necessarilyaddress any of these deficiencies. In other words, different embodimentsof the invention may address different deficiencies that may bediscussed in the specification. Some embodiments may only partiallyaddress some deficiencies or just one deficiency that may be discussedin the specification, and some embodiments may not address any of thesedeficiencies.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following drawings like reference numbers are used to refer tolike elements. Although the following figures depict various examples ofthe invention, the invention is not limited to the examples depicted inthe figures.

FIG. 1 illustrates a block diagram of example of functional componentsof the method and system for scoring and ranking articles;

FIG. 2 illustrates a block diagram of an embodiment of a document foruse with the method and system for scoring and ranking articles;

FIGS. 3A-3B are operational flow diagrams illustrating an embodiment ofa method of scoring and ranking of articles;

FIG. 4 is a flow diagram illustrating embodiments of scoring and rankinga knowledge article;

FIG. 5 illustrates a block diagram of an example of an environmentwherein an on-demand database service might be used;

FIG. 6 illustrates a block diagram of an embodiment of elements of FIG.5 and various possible interconnections between these elements;

FIG. 7 is a flowchart of an example of a method of using the environmentin FIGS. 5 and 6;

FIG. 8 is an embodiment of method of making the environment in FIGS. 5and 6.

DETAILED DESCRIPTION General Overview

Systems and methods are provided for scoring and ranking in an on-demandservice environment. The following detailed description will firstdescribe exemplary functional components for scoring and rankingarticles in an on-demand services environment in accordance with aspectsand embodiments of the present invention. A block diagram of a documentis then detailed.

As used herein, the term multi-tenant database system refers to thosesystems in which various elements of hardware and software of thedatabase system may be shared by one or more customers. The multitenantdatabase may be a relational database may have multiple tenants. Each ofthe tenants may be an organization and/or may have their own customers,representatives, and/or employees whom may share and/or benefit from theresources of the multitenant database system, and/or whom may be usersof the multitenant database as a result of the organization be a tenantof the multitenant database. For example, a given application server maysimultaneously process requests for a great number of customers, and agiven database table may store rows for a potentially much greaternumber of customers. As used herein, the term query plan refers to a setof steps used to access information in a database system. Usage of thetenant's portion of multitenant database system may be provided to thetenant on-demand—that is as a service, so that the tenant does not needto be concerned with managing the database system and/or withoutrequiring the tenant to download software related to the multitenantdatabase.

Next, mechanisms and methods for scoring and ranking in an on-demandservice environment will be described with reference to exampleembodiments. The methods involve deriving a score based on a variety ofuser actions in a forum and/or when processing a search, then rankingthe knowledge article. The ranking decides what order a knowledgearticle appears in a results set returned in response to a usersubmitting a search.

Next, mechanisms and methods for providing methods and systems forscoring and ranking articles identified using search terms (e.g.,keywords) will be described with reference to example embodiments. Inthis specification a community is a group of users that share a commoninterest and tend to communicate with one another about that commoninterest.

A community website is a website in which users share ideas that arerelated to the community, which may include at least a portion of one ormore webpages. An on-demand community is a community website that iscreated by the tenant using pre-established tools specialized forcreating communities. The multitenant database system may provide thepre-established tools specialized for creating the communities. Usingthe pre-established tools, the community website and the ability toestablish a community website may be provided to the tenant of themultitenant database as a service to the tenant that is managed by themultitenant database. The tenant of the multitenant database may notneed to download any software to establish and run the community websiteprovided by the multitenant database system. In an embodiment, the focusof the communities may be on-demand enterprise communities, which arecommunities structured around a business' partners, customers, vendors,representatives, employees, and/or others associated with the business.

A knowledge base is a database for knowledge management. The knowledgebases can be part of a community website. The knowledge base is acollection of articles related to problems and solutions related to thecommon interest of the community. Knowledge base systems may includetools that help in creating, editing, and managing the database ofarticles. Knowledge base systems may offer documentation of knowledgeand self-learning there by reducing the cost of customer support. Thearticles in the knowledge base may include some problems and solutionsthat were initially discussed in the forum, were deemed (e.g., by anadministrator) to be of particular importance to users, and wereconsequently copied from the log of questions and answers of the forumto the knowledge base (and optionally further formatted and/or otherwiseenhanced for clarity and/or to provide a more thorough discussion of thetopic).

Analytics in this specification refers to the science of analysisconcerned with extracting useful properties of data, typically involvingextracting properties from large databases. Search analytics in thisspecification refers to the analysis and aggregation of search enginestatistics. Search analytics may help in understanding and/or improvingthe performance of search engines. Search analytics may includedetermining search volume trends, analysis of searches, keywords,advertisements, search volume trends, keyword monitoring, searchresults, advertisement history, advertisement spending statistics,and/or comparisons of websites, for example. The comparison of websitesmay include a comparison of the number and/or type of users,advertisers, advertisements, keywords, search results, search trends,advertisement trends, and/or advertisement histories, for example.

In this specification a keyword refers to a word or a phrase relevant toa webpage document, via which the document may be found. Keyword canalso be a word or a phrase entered in a search engine. In thisspecification, the term document is generic to a webpage, an article, orother document. A page of results (which may be a webpage) from akeyword search in a search engine may also be referred to as searchengine results page. Keywords and search terms are used interchangeablyin this specification and may be substituted for one another to obtaindifferent embodiments. In this specification the term keyword searchanalytics refers to the analysis and aggregation of search engine data.Keyword search analytics can be helpful in understanding the trends andrequirements of users which in turn helps in optimizing the database.

The application platform is a platform for hosting a forum about aproduct or customer related topic. Application platform facilitates theorganizing of the questions and answers presented during the forum in amanner so that the information receives appropriate attention. In anembodiment, the forum may be a community website and may have a commongoal of discussing a product. Associated with the forum may be severaldifferent storage areas (which may each include one or more tablesand/or databases). The storage areas may include one storage area forquestions and answers, one for customer case management, and one forproblems and solutions, for example. The questions and answers storagearea may be a log and/or database storing the questions and answers ofthe forum. The customer case management may store a list of problemsthat customers are currently having and/or that need to be solved. Thecustomer case management system may store information about specificissues that need to be resolved for specific clients, such as whichissues have troubled which clients, how long ago the issue appeared,whether any action has been taken to resolve the problem worked. Theproblems and solutions may include problems that customers had and howthe problems were solved. The knowledge base may be a database ofknowledge articles. Generally, knowledge articles may be entered bycustomer support agents. The terms “knowledge articles” and “articles”are used interchangeably in this specification and may includeinformation that customers, customer care agents, and/or administratorsare expected to find useful. The knowledge articles may come from theforum, but optionally knowledge articles may also come from elsewhere.For example, knowledge articles may include articles having informationabout a new product, tutorials for using a product, and/or otherspecific topics, may be placed in the knowledge base of the forum by thedeveloper at the time that a produce it released. Once a question ispresented to the forum, the question is visible to other users, and theother users can reply to the posted question. If a particular questionsis troubling a user, the user has the option of browsing for similarquestions to the user's questions or presenting the question to theforum. Apart from answering questions, internal and external users canvote on whether they like the posted question and/or answers (internalusers are users that are members of the organization running themultitenant database and external users are users that are not membersof the organization running the database, such as members oforganizations that are tenants of the multitenant database). If anexternal user (such as a member of an organization that is a tenant)votes on a posted question or answer, the vote would be referred to asan external vote.

If authorized personnel (that is personnel authorized by theorganization running the multitenant database) considers that aparticular question has been well addressed, the authorized personnelmay vote (and this vote would be an internal votes).

Knowledge Article Scoring

Particular embodiments include an approach that assigns scores to a“knowledge article” in order to support ranking of that article insearch results. The score provides an indication of the quality oroverall usefulness of an article, but does not measure the relevancy ofthe article to any particular question or search. In this specification,the quality of the article may refer to the likelihood that others wouldbe interested in the article. When a search query is received one factorin determining the ranking of the article in search results may be thescore and another factor determining the ranking may be the relevancy.Thus, for example, the ranking of the article may be determined byadding the score times a first weight to the relevancy times a secondweight. Knowledge scoring may include various parameters, such as arating system, the number of reference to the article being scored inother knowledge articles, the number of references to the article beingscored in the cases and solutions storage area, on the number externaluser visitation to the article being scored, the number of internal uservisitations to the article being scored, and/or the age of the article,the votes, the references to the article, and/or the visitations.

Each article may be rated according to a rating system, such as a 5-starrating system. In an embodiment, this 5-star rating may be a primarymethod of weighting in scoring. If the article is exposed on publicsites, and if voting is allowed in the public site, then the votes bypublic users (external users) may also be assigned a weight (whichweights the contribution of the external votes to the score). Rating byinternal users may also be assigned an appropriate weight. Externalvotes can reflect the article's score from a public user perspective,which may be considered more important since the external users may bethe major consumers of the articles. As such, particular embodiments mayassign a higher weight to the votes of external users.

Knowledge articles may also be able to refer to other knowledgearticles. Such references aid in keeping the content of a knowledgearticle focused to a particular topic, and also can avoid repetition ofinformation that is also available in other articles. If a knowledgearticle “A” is referenced in another article “B,” article “A” can thusget additional scores with a predetermined weight. In an embodiment, thereferring article's score can also be considered as a parameter inparticular embodiments.

Knowledge articles may be referenced in a storage area for cases andsolutions (and likely in other entities, such as a storage are forquestions and replies that occurred as part of the forum). Whenever anarticle gets referenced in the cases and solutions storage area, thescore of the knowledge article may be incremented by an amount thatreflects the expected significance of the reference in indicating thequality of the article (or the degree to which it is expected thatothers will be interested in the article). In an embodiment, theincrement to the score may be the increase in the number of referencesto the article times a weight.

Also, in particular embodiments, a weight may be assigned based on anumber of times an article is visited by external users. Such aweighting may be different than the rating discussed above. For example,a visit to a knowledge article may not be considered as significant as avote indicating that the voter liked the article. Nevertheless, articleswhich are read, may receive increments to the scores of those articlesbased on the number of times the articles are read.

In addition, in an embodiment, a “half-life” may be assigned to votesand/or other increments to the score (similar to the half-life used incomputing scores of “ideas”) so that newer and active articles may tendto receive higher scores than older, inactive articles (see U.S. patentapplication Ser. No. 11/716,365, filed Mar. 8, 2007, herein incorporatedby reference in its entirety).

A final score of the article can then be computed based on one or moreof the above parameters, as well as any suitable additional parameters,and may reflect the likelihood that users will be interested in anarticle.

In particular embodiments, a score of an article can be maintained as aseparate field in the article itself (e.g., in the meta data of thearticle), and the description of the knowledge article in the universaldata description (UDD) may include a field for the score. In otherwords, there may be a library of templates for different types ofdocuments and/or other objects, which may include the different fieldsthat are to be included in each type of document or object in thelibrary, and the template for the knowledge article may include a fieldfor the score.

Particular embodiments may also be enhanced to automatically detect thereferences to articles in other articles, in a storage area for casesand solutions, and/or to keep a count of the number of times the articlewas read and/or visited. Further, different parameters can beconfigurable by the tenant organization and/or a module deployed (e.g.,over the Internet) associated with the web community of the forum.

FIG. 1 illustrates a block diagram of an embodiment of a scoring andranking system 100. In an embodiment, the functional components 100 mayinclude: forum 141, knowledge base 145, questions and answers 150, casesand solutions 155, algorithm for computing rank 160, vote table 165,algorithm for computing score 170, cross reference tracker 180, and readcounter 190. In other embodiments, functional components 100 may nothave all of the elements or features listed and/or may have otherelements or features instead of or in addition to those listed.

Each block is a functional component of the system for scoring andranking knowledge articles. Scoring and ranking system 100 may computethe score and/or rank of an article. The forum 141 includes thediscussion between users in the forum 141. This can include but is notlimited to, online discussions, posting of questions, posting ofanswers, posting of solutions, posting of opinions, etc. The forum canbe about a product or a customer related topic.

The knowledge base 145 is a database for knowledge management. Knowledgebase 145 can be part of a community website. Knowledge base 145 is acollection of articles related to problems and solutions related to thecommon interest of the community knowledge base 145 may include one ormore databases (e.g., are relational database) where the articles arestored. Knowledge base 145 may be included in a system that has toolsthat help in creating, editing, and managing the database of articlesstored in knowledge base 145. The system in which knowledge base 145 isstored may offer documentation of knowledge and self-learning therebyreducing the cost of customer support. Optionally, knowledge base 145may have varying levels of access for different users and/or fordifferent types of users, such that some content is not available if theuser does not have authorization, but other content is available to theuser.

Questions and answers 150, is a storage area that may store a log of theforum and can be open to the public for searching or may only be partlysearchable by the public (there may be different levels of accessassociated with different content stored in questions and answers 145).In other words, optionally, questions and answers 150 may have varyinglevels of access for different users and/or for different types ofusers, such that some content is not available if the user does not haveauthorization, but other content is available to the user. Questions andanswers 150 may be a relations database or other database, which may bea part of, or separate from, knowledge base 145. Optionally, thequestions and answers 150 can also include calls or emails. The forumparticipants may rate how much the participants liked a question oranswer. The rating system may be use a “5 star rating system.” Someonereviews the questions of the questions and answers. The questions andanswers that are believed to be beneficial for others to read are placedinto the knowledge base. In an embodiment, other articles in theknowledge base that relate to the topic of interest of the knowledgearticle.

Cases and solutions 155 is a storage area that may include a record ofcustomer complaints and the solution arrived at for that customer. Casesand solutions 155 may be a relational database or other database, whichmay be a part of, or separate from, knowledge base 145. Cases andsolutions 155 can be handled by a representative of the company. Therepresentative can create a case when there is a question from a user,which may be updated when there is a solution. Optionally, cases andsolutions 155 may have varying levels of access for different usersand/or for different types of users, such that some content is notavailable if the user does not have authorization, but other content isavailable to the user.

The algorithm for computing rank 160 may include an algorithm fordetermining the order in which articles are presented to a user in a setof results presented in response to a search. The search engine may rankthe knowledge article according to the search term and the relevance.The rank is determined by a combination of the relevance to a particularsearch (or keyword) and the score.

The vote table 165 may be a table of votes received for a variousarticles. For example, the table may include one column for document IDsand another column for the number of votes received by the articlehaving that document ID. In an embodiment, voting information is thedata related to voting of articles, which might include a normalizedscore, channel, total sum, total number of votes, and/or the number of 1star, 2 stars, 3 stars, 4 stars, and/or 5 stars votes. User systems maybe capable of accessing the knowledge base in many ways and channelinformation indicates the access path to the articles in the knowledgebase. In some embodiments, voting involves rating an article from 1 to 5stars, based on the user system sending a vote liking an article and/orthe user system sending an indication as to how many stars to award tothe idea. An article that has the highest degree of being liked may beawarded a 5 star vote and least liked may receive a 1 star vote. In anembodiment, an article which receives a 5 star vote from the user systemis given 10 points, an articles which receives a 4 star vote from theuser system is given 5 points, an article which receives a 3 star votefrom the user system is given 0 points, an article which receives a 2star vote from the user system is given −5 points, and an article whichreceives a 1 star vote from the user system is given −10 points. Thenormalization of setting what is expected to be an average value (threestars) to the value 0, and assigning negative values to values below theexpected average (e.g., −5 for two stars and −10 for one star) andpositive values to values that are expected to be above the expectedaverage value (e.g., 5 for four stars and 10 for five stars) results inthe total of the scores from several voters being a lower number, andless likely to cause an overflow error. The total votes are the totalnumber of votes an article has received since the initial posting of thearticle in the forum. The votes may be weighted according to howrecently the votes were cast (with more recent votes receiving a higherweight). The vote of external users may be weighted more than internalusers. In one embodiment, vote table 165 stores external and internalvotes separately (e.g., in different columns), and in anotherembodiment, only the combination of the external and internal votes arestored. The vote table 165 can include database keys (in addition to orinstead of the document ID) to link to other tables. Vote table 165 isoptional. The score could be updated with each vote, so that there is noneed to store the actual votes.

The algorithm for computing a score 170, can take into account all ofthe aspects of the score (which will be discussed further below in FIG.3A) including but not limited to, voting of external and internal users,number of times the knowledge article is visited, reference to otherknowledge articles, and age. The score can help to determine the rankingof the knowledge article. The score is not related to a particularsearch, but a measure of how useful the article is generally expected tobe. In an embodiment, the score is updated every time an event occursthat affects the score (e.g., every time someone votes on the article,visits the article, adds a reference to the article in another article).In an alternative embodiment, the score is re-determined at a specifictime, e.g, every 30 days, every 7 days, etc. The algorithm fordetermining the score can be a weighted average. For example the weightfor each attribute X the number of such scores is the weighted sum. Thiscan be combined with the decay in time as a way of weighting the votes.The decay in time can be a half-life period, including but not limitedto, 10 days if an idea is not being viewed.

In an embodiment, the algorithm computes the score by performing aseries of steps that yield a result that is equivalent to or similar tothe results of computing the formula,

s=Σ_(i=1) ^(N)e^(−αt) ^(i) Σ_(j=1) ^(M)w_(j)F_(j)(t_(i)), where s is thescore, N is the total number of times at which the time score wasincremented, i is an index labeling the times at which score wasincremented, t_(i) is the time at the ith time that the score wasincremented, M is the total number factors that are used in computingthe score, j is an index labeling each factor used to compute the score,F_(j) is the jth factor used to compute the score, and w_(j) is theweight assigned to factor F_(j). The weights w_(j) are numerical values.The magnitude of any particular weight w_(j) is not important, but theration of the each weight with respect to one another determines therelative percentage that factor the weight is associated with isassigned.

One computation that may be performed that is equivalent to the aboveformula is to compute the score from the following recursive formula,

s(t_(i))=K(t_(i))Δ(t_(i))+s(t_(i-1)), where s(t_(i-1)) is the priorscore computed prior to the most recent increment of the score, whichwas update at the prior time t_(i-1), Δ(t_(i)) is the current incrementof the score, which is being incremented at time t₁, which may becomputed from Δ(t_(i))=Σ_(j=1) ^(M)w_(j)F_(j)(t_(i)), K(t_(i)) is aconstant that is incremented essentially every time the score isupdated, which is related to the decay constant α, and s(t_(i)) is thecurrent score being computed at time t_(i). In an embodiment, Δ(t_(i))may be computed from the formulaΔ(t_(i))=w_(vi)V_(int)(t_(i))+w_(vx)V_(ex)(t_(i))+w_(R)R(t_(i))+w_(c)C(t_(i)),where C(t_(i)) is the number of cross references to the current articlereceived at time t_(i), w_(c), is the weight assigned to the number ofcross references, R(t_(i)) is the number of reads or visitations thecurrent article received at time t_(i), w_(R) is weight assigned to thenumber of reads or visitations to the current article, V_(ex)(t_(i)) isthe number of external votes received for the current article receivedat time t_(i), w_(vx) is weight assigned to the number external votesfor the current article, V_(int)(t_(i)) is the number of internal votesfor the current article received at time t_(i), and w_(vi) is the weightassigned to the number of internal votes for the current article. Toavoid an overflow error, periodically, all of the scores are divided bya large number. In an embodiment, K(t_(i)) may be computed recursivelyaccording to the formula (t_(i))=βK(t_(i-1)), where K(t_(i-1)) is theprior constant and β is a positive constant greater than 1. Theconstants β and α are phenomenological constants that are determinedexperimentally according to the users needs, volume and/or frequency ofupdates to the score. Consequently, the relationship between β and α isnot important—the appropriate value to use for β may be determinedexperimentally without knowledge of α of the specific half life itcorresponds.

In an embodiment w_(vx) should be greater than w_(vi) so that externalvotes are given more weight than internal votes.

Cross reference tracker 180 is an algorithm that tracks cross-referencesbetween knowledge articles. The cross-reference tracker can keep trackof how many times an article is referenced by other articles. In anembodiment, cross reference tracker 180 detects the creation of a crossreference to an article. In an embodiment, after detecting the creationof a cross reference, cross reference tracker initiates an update of thescore for the article and/or updates a counter that tracks the number ofcross references to the article.

Read counter 190 measures how many people visit and/or read the article.The read count may be stored in the article itself. The read count maybe included in the score. The read count will increase each time anarticle is opened and/or read.

FIG. 2 illustrates a block diagram of an embodiment of a document 200for the method and system for scoring and ranking articles. In anembodiment, the document 200 may include meta data 215, score 220, readcount 225, other fields 230, and content 240. In other embodiments,document 200 may not have all of the elements or features listed and/ormay have other elements or features instead of or in addition to thoselisted.

In an embodiment, document 200 is a software version of a document. Thedocument has a text, but also has a something that defines where thepage begins and ends and how to open and read.

Meta data 215 includes at least the score 220, the read count 225, andother fields 230. The meta data 215 can include machine instructions(e.g. code) of how to open and read (e.g. render) and a definition forwhere the page begins and ends. In an embodiment, the meta data isreadily visible in a view that intended for developers (as opposed toend users), but is not readily visible in a view that is intended forend users. The meta data may also include the score.

The score 220 of the knowledge article may be stored in the document aspart of the meta data 215, which may aid in ranking the knowledgearticle.

The read count 225 may be stored in meta data 215 and describes thenumber of times the knowledge article has been read, opened, and/orvisited.

Other Fields 230 includes other meta data that is included in thedocument 200 (in addition to the score 220 and the read count 225). Forexample, other fields 230 may include a counter that tracks the numberof cross references that have been created to the current article.

The content 240 is separate from the meta data 215, and may include thecontent of the knowledge article, which may include text and/or images,for example. The content 240 can include, but is not limited to, text,discussions, solutions, questions, conversations from the forum, and/orother content. Content 240 may placed in a location of the document thatis readily visible to an end user. Content 240 and metadata 215 may bestored in different locations in a database where document 200 isstored. The system may store machine instructions for placing thecontent and in the content area of document 200 and the meta data in themeta data area of document 200 based on which are of the database theinformation is stored. In other words, information in the content areaof the database may automatically be placed in the content area of thedocument upon being rendered, and information in the meta data area ofthe database may automatically be placed in the meta data area of thedocument upon being rendered.

Methods of Scoring and Ranking Knowledge Articles (FIGS. 3A, 3B and 4)

FIGS. 3A-3B show operational flow diagrams illustrating scoring andranking of articles.

FIG. 3A shows a flowchart of an example of a method of receiving asearch 300, carried out by the host system, and shows how the user caninfluence the score of the knowledge article identified by the search(e.g., the user can vote and/or can influence the score by viewing thearticle).

In step 302, a search is received from a user. The search can be anytype including a keyword search, a question, and/or a complaint.

In step 303, an initial set of articles is determined that match thesearch to at least some degree (which therefore have a certain degree ofrelevance). For example, the initial set of articles may be a set ofarticles that have one or more of the search terms.

In step 304, a ranking is determined for each of the articles found thatare in the initial set of articles. The ranking may be a weighted sum ofrelevancy factors and the score. For example, a certain value may beadded to the ranking for each of key word of the search term that isfound in the article. The amount that is added to the ranking may dependon the number of times the word is found in the article and/or the ratioof the number of time the terms is found in the article to number ofwords in the article. An additional amount may be added to the rankingfor each of the words of the search term found in the abstract. Anadditional amount may be added to the ranking for each of the words ofthe search term found in the title. If article is opened by a user, theread counter is updated, the score recomputed, and the score of thearticle is updated and stored in the meta data of the score.

In step 306, the results are placed in order of decreasing rank,starting with the article with the highest rank and placed into a pageof results that is sent to the user. The user may choose to open one ormore of the knowledge articles in the results. By opening the article orarticles, the score of the article or articles may be changed (see FIG.4).

In step 308 a determination is made as to whether a message has beenreceived that the article has been opened and/or read. If the article isopened and/or read, the score is updated. If the article is not opened,the method ends.

In step 310, the score is updated based on the opening and/or reading ofan article. The score can be updated as discussed in FIG. 4 (see step404).

In step 312, a determination is made as to whether a vote has beenreceived from a user. If a vote has been received, the score is updated.If a vote has not been received, the method ends.

In step 314 the score is updated because a vote is received from anexternal and/or internal user. External votes (from users) may beweighted greater than internal votes. External users are users that areassociated with tenants of the multitenant database system. Users mayalso include internal users, public users and/or forum members. Internalusers may be users associated with an organization that runs themultitenant database system (see also FIG. 4, step 402, which isdiscussed below).

In step 316 a new score of the knowledge article can be determined basedon the new vote (see also FIG. 4).

In another embodiment, although depicted as distinct steps in FIG. 3A,steps 304-310 may not be distinct steps. In other embodiments, method300 may not have all of the above steps and/or may have other steps inaddition to, or instead of, those listed above. The steps of method 300may be performed in another order. Subsets of the steps listed above aspart of method 300 may be used to form their own method.

FIG. 3B shows a flowchart of an example of a user search 350, carriedout by the user system, and how the actions of the user can influencethe score and/or ranking of a knowledge article. The methods of FIGS. 3Aand 3B may be combined to form two parts of the same method, where themethod FIG. 3A is the portion of the combined method carried out by thehost system and the method of FIG. 3B is the portion of the combinedmethod carried out by the user's system.

In step 352, a user sends a search. The search can be any type includinga keyword (or term) search, a question, and/or a complaint. Theknowledge articles may be searched by identifying those that are mostrelated to the search term. The search of step 352 of FIG. 3B may bereceived in step 302 of FIG. 3A.

In step 354, a web page with the ranked articles is received, which maybe the result of the ranked articles being sent in step 306 of FIG. 3A.The articles are ranked based on how they fit the search as well as thescores the articles have received. Thus, the method allows for thedirect input of one or more users to determine the relevancy andauthenticity of an article and to solve particular problems. The userhelps to solve answers and scoring is partly or mostly determined by theend-user.

In step 356, a user decides whether to open and/or read and article. Ifthe article is opened and/or read, a message is sent to the server andthe score is updated. If the article is not opened, the method ends.

In step 358, if a user opens an article, and the user machine may send amessage to the host machine that the article has been opened, whichinitiates step 308 of FIG. 3A. By opening the article, the userinfluences the score of the article, by causing step 308 to beperformed. The read count of an article increases each time the articleis opened and/or read.

In step 360, a user decides whether to vote on an article. If the uservotes on the article, a vote is sent to the server. If the user does notvote on the article, the method ends.

In step 362 if the user votes, a vote is sent to the server. The scoringof a particular article can be triggered each time a vote occurs. Eacharticle can include a rating system, such as a 5-star rating system.This 5-star rating may be a primary method of weighting scores. If thearticle is exposed on public sites, and if voting is allowed in thepublic site, then the votes by public users may also bear an attachedweight thereto. Rating by internal users may also carry an appropriateweight. External votes can reflect the articles score from a public userperspective, which may be considered more important since the externalusers are the major consumers of the articles. As such, particularembodiments can provide higher weight to the votes of external users.

In another embodiment, although depicted as distinct steps in FIG. 3B,steps 352-358 may not be distinct steps. In other embodiments, method350 may not have all of the above steps and/or may have other steps inaddition to, or instead of, those listed above. The steps of method 350may be performed in another order. Subsets of the steps listed above aspart of method 350 may be used to form their own method.

FIG. 4 shows a flowchart of an example of scoring and ranking aknowledge article 400 by first computing a score and then ranking theknowledge article.

In step 402, the knowledge article is scored based on voting. Thescoring of a particular article can be triggered each time a voteoccurs. Each article can include a rating system, such as a 5-starrating system. This 5-star rating may be a primary method of weightingin scoring. If the article is exposed on public sites, and if voting isallowed in the public site, then the votes by public users may also bearan attached weight thereto. Rating by internal users may also carry anappropriate weight. External votes can reflect the articles score from apublic user perspective, which may be considered more important sincethe external users are the major consumers of the articles. As such,particular embodiments can provide higher weight to the votes ofexternal users.

In step 404, the knowledge article is scored based on the number oftimes the article is visited, which may be recorded in the rate countervalue. Also, in particular embodiments, a weight may be assigned basedon the number of times an article is visited. For example, a visit to aknowledge article may not be considered as significant as a voting.Nevertheless, articles which are read, can receive scores based on thenumber of times the articles are read. The number of times the articleis read or visited is taken into account and used to update the scoreeach time the score is updated and/or each time the article is visitedand/or read.

In step 406, a knowledge article is scored based on reference to otherknowledge articles. Knowledge articles may also be able to refer toother knowledge articles. Cross references from one article to anotheraid in keeping the content of a knowledge article focused to aparticular topic, and also can avoid repetition of information that isalso available in other articles. If a knowledge article “A” isreferenced in another article “B”, article “A” can thus get additionalscores with predetermined weight. In an embodiment, the referringarticle's score can also be considered as a parameter in particularembodiments. The cases and solutions and other questions and answers cantrigger a new score. Links can be put in knowledge articles to refer toother knowledge articles. The links can be a label of the knowledgearticle that can be used by outside users to uniquely identify knowledgearticles so that when on article references another, the knowledgearticle can easily be found. Cross referencing can be tracked as thecross references are added to the article so that the cross referencewill be added to the score of the article. Thus, each time the articleis referenced, a new score can be determined. The links may also be away for managers to see and follow activities related to the usage of acustomer support system. The links can be used for the manager to seethe evolution of activities over time.

In step 408, a knowledge article is scored based on age. In addition,each increment to the score may be associated with a “half-life,” sothat newer activities that cause the score to be incremented may receivea higher score than older activities of the same type. For example, morerecent votes may have a higher weight than older votes, more recentvisits may have a higher weight than visits votes, and/or more recentcross references may have a higher weight than older cross references.The order of steps 402-406 is not important, but they must be finishedbefore the knowledge article can be scored based on age. The score basedon age can be computed as a weighted sum. The weighted sum can be basedon the weight given to each vote as well as the time.

In step 410, a new score of the knowledge article is determined. A finalscore of the article can then be computed based on one or more of theabove parameters, as well as any suitable additional parameters, and mayreflect the ranking of the article in the search results. Each time asearch is performed, the relevancy of the article to the search terms isdetermined and is combined with the score to form a ranking of thearticle.

In another embodiment, although depicted as distinct steps in FIG. 4,steps 402-410 may not be distinct steps. In other embodiments, method400 may not have all of the above steps and/or may have other steps inaddition to, or instead of, those listed above. The steps of method 400may be performed in another order. Subsets of the steps listed above aspart of method 400 may be used to form their own method.

System Overview

FIG. 5 illustrates a block diagram of an environment 510 wherein anon-demand database service might be used. Environment 510 may includeuser systems 512, network 514, system 516, processor system 517,application platform 518, network interface 520, tenant data storage522, system data storage 524, program code 526, and process space 528.In other embodiments, environment 510 may not have all of the componentslisted and/or may have other elements instead of, or in addition to,those listed above.

Environment 510 is an environment in which an on-demand database serviceexists. User system 512 may be any machine or system that is used by auser to access a database user system. For example, any of user systems512 can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Asillustrated in FIG. 5 (and in more detail in FIG. 6) user systems 512might interact via a network 514 with an on-demand database service,which is system 516.

An on-demand database service, such as system 516, is a pre-establisheddatabase system that is made available to outside users that do not needto necessarily be concerned with building and/or maintaining thedatabase system, but instead may be available for their use when theusers need the database system (e.g., on the demand of the users). Someon-demand database services may store information from one or moretenants stored into tables of a common database image to form amulti-tenant database system (MTS). Accordingly, “on-demand databaseservice 516” and “system 516” will be used interchangeably herein. Adatabase image may include one or more database objects. A relationaldatabase management system (RDMS) or the equivalent may execute storageand retrieval of information against the database object(s). Applicationplatform 518 may be a framework that allows the applications of system516 to run, such as the hardware and/or software, e.g., the operatingsystem. In an embodiment, on-demand database service 516 may include anapplication platform 518 that enables creation, managing and executingone or more applications developed by the provider of the on-demanddatabase service, users accessing the on-demand database service viauser systems 512, or third party application developers accessing theon-demand database service via user systems 512.

The users of user systems 512 may differ in their respective capacities,and the capacity of a particular user system 512 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 512 tointeract with system 516, that user system has the capacities allottedto that salesperson. However, while an administrator is using that usersystem to interact with system 516, that user system has the capacitiesallotted to that administrator. In systems with a hierarchical rolemodel, users at one permission level may have access to applications,data, and database information accessible by a lower permission leveluser, but may not have access to certain applications, databaseinformation, and data accessible by a user at a higher permission level.Thus, different users will have different capabilities with regard toaccessing and modifying application and database information, dependingon a user's security or permission level.

Network 515 is any network or combination of networks of devices thatcommunicate with one another. For example, network 514 can be any one orany combination of a LAN (local area network), WAN (wide area network),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a TCP/IP (Transfer Control Protocol and Internet Protocol)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I,” that network will be used in manyof the examples herein. However, it should be understood that thenetworks that the present invention might use are not so limited,although TCP/IP is a frequently implemented protocol.

User systems 512 might communicate with system 516 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTPis used, user system 512 might include an HTTP client commonly referredto as a “browser” for sending and receiving HTTP messages to and from anHTTP server at system 516. Such an HTTP server might be implemented asthe sole network interface between system 516 and network 514, but othertechniques might be used as well or instead. In some implementations,the interface between system 516 and network 514 includes load sharingfunctionality, such as round-robin HTTP request distributors to balanceloads and distribute incoming HTTP requests evenly over a plurality ofservers. At least as for the users that are accessing that server, eachof the plurality of servers has access to the MTS' data; however, otheralternative configurations may be used instead.

In one embodiment, system 516, shown in FIG. 5, implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 516 includes application servers configured toimplement and execute CRM software applications as well as providerelated data, code, forms, webpages and other information to and fromuser systems 512 and to store to, and retrieve from, a database systemrelated data, objects, and Webpage content. With a multi-tenant system,data for multiple tenants may be stored in the same physical databaseobject, however, tenant data typically is arranged so that data of onetenant is kept logically separate from that of other tenants so that onetenant does not have access to another tenant's data, unless such datais expressly shared. In certain embodiments, system 16 implementsapplications other than, or in addition to, a CRM application. Forexample, system 16 may provide tenant access to multiple hosted(standard and custom) applications, including a CRM application. User(or third party developer) applications, which may or may not includeCRM, may be supported by the application platform 518, which managescreation, storage of the applications into one or more database objectsand executing of the applications in a virtual machine in the processspace of the system 516.

One arrangement for elements of system 516 is shown in FIG. 5, includinga network interface 520, application platform 518, tenant data storage522 for tenant data 523, system data storage 524 for system data 525accessible to system 516 and possibly multiple tenants, program code 526for implementing various functions of system 516, and a process space528 for executing MTS system processes and tenant-specific processes,such as running applications as part of an application hosting service.Additional processes that may execute on system 516 include databaseindexing processes.

Several elements in the system shown in FIG. 5 include conventional,well-known elements that are explained only briefly here. For example,each user system 512 could include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 512 typically runs an HTTP client, e.g., abrowsing program, such as Microsoft's Internet Explorer browser,Netscape's Navigator browser, Opera's browser, or a WAP-enabled browserin the case of a cell phone, PDA or other wireless device, or the like,allowing a user (e.g., subscriber of the multi-tenant database system)of user system 512 to access, process and view information, pages andapplications available to it from system 516 over network 514. Each usersystem 512 also typically includes one or more user interface devices,such as a keyboard, a mouse, trackball, touch pad, touch screen, pen orthe like, for interacting with a graphical user interface (GUI) providedby the browser on a display (e.g., a monitor screen, LCD display, etc.)in conjunction with pages, forms, applications and other informationprovided by system 516 or other systems or servers. For example, theuser interface device can be used to access data and applications hostedby system 516, and to perform searches on stored data, and otherwiseallow a user to interact with various GUI pages that may be presented toa user. As discussed above, embodiments are suitable for use with theInternet, which refers to a specific global internetwork of networks.However, it should be understood that other networks can be used insteadof the Internet, such as an intranet, an extranet, a virtual privatenetwork (VPN), a non-TCP/IP based network, any LAN or WAN or the like.

According to one embodiment, each user system 512 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Pentium® processor or the like. Similarly, system 16(and additional instances of an MTS, where more than one is present) andall of their components might be operator configurable usingapplication(s) including computer code to run using a central processingunit such as processor system 517, which may include an Intel Pentium®processor or the like, and/or multiple processor units. A computerprogram product embodiment includes a machine-readable storage medium(media) having instructions stored thereon/in which can be used toprogram a computer to perform any of the processes of the embodimentsdescribed herein. Computer code for operating and configuring system 516to intercommunicate and to process webpages, applications and other dataand media content as described herein are preferably downloaded andstored on a hard disk, but the entire program code, or portions thereof,may also be stored in any other volatile or non-volatile memory mediumor device as is well known, such as a ROM or RAM, or provided on anymedia capable of storing program code, such as any type of rotatingmedia including floppy disks, optical discs, digital versatile disk(DVD), compact disk (CD), microdrive, and magneto-optical disks, andmagnetic or optical cards, nanosystems (including molecular memory ICs),or any type of media or device suitable for storing instructions and/ordata. Additionally, the entire program code, or portions thereof, may betransmitted and downloaded from a software source over a transmissionmedium, e.g., over the Internet, or from another server, as is wellknown, or transmitted over any other conventional network connection asis well known (e.g., extranet, VPN, LAN, etc.) using any communicationmedium and protocols (e.g., TCP/IP, HTTP, HTTPS, Ethernet, etc.) as arewell known. It will also be appreciated that computer code forimplementing embodiments of the present invention can be implemented inany programming language that can be executed on a client system and/orserver or server system such as, for example, C, C++, HTML, any othermarkup language, Java™, JavaScript, ActiveX, any other scriptinglanguage, such as VBScript, and many other programming languages as arewell known may be used. (Java™ is a trademark of Sun Microsystems,Inc.).

According to one embodiment, each system 516 is configured to providewebpages, forms, applications, data and media content to user (client)systems 512 to support the access by user systems 512 as tenants ofsystem 516. As such, system 516 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB). As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., OODBMS or RDBMS) as is well known in the art. It should also beunderstood that “server system” and “server” are often usedinterchangeably herein. Similarly, the database object described hereincan be implemented as single databases, a distributed database, acollection of distributed databases, a database with redundant online oroffline backups or other redundancies, etc., and might include adistributed database or storage network and associated processingintelligence.

FIG. 6 also illustrates environment 510. However, in FIG. 6 elements ofsystem 516 and various interconnections in an embodiment are furtherillustrated. FIG. 6 shows that user system 512 may include processorsystem 512A, memory system 512B, input system 512C, and output system512D. FIG. 6 shows network 514 and system 516. FIG. 2 also shows thatsystem 516 may include tenant data storage 522, tenant data 523, systemdata storage 524, system data 525, User Interface (UI) 630, ApplicationProgram Interface (API) 632, PL/SOQL 634, save routines 636, applicationsetup mechanism 638, applications servers 1000 ₁-1000 _(N), systemprocess space 602, tenant process spaces 604, tenant management processspace 610, tenant storage area 612, user storage 614, and applicationmetadata 616. In other embodiments, environment 510 may not have thesame elements as those listed above and/or may have other elementsinstead of, or in addition to, those listed above.

User system 512, network 514, system 516, tenant data storage 522, andsystem data storage 524 were discussed above in FIG. 5. Regarding usersystem 512, processor system 512A may be any combination of one or moreprocessors. Memory system 512B may be any combination of one or morememory devices, short term, and/or long term memory. Input system 512Cmay be any combination of input devices, such as one or more keyboards,mice, trackballs, scanners, cameras, and/or interfaces to networks.Output system 512D may be any combination of output devices, such as oneor more monitors, printers, and/or interfaces to networks. As shown byFIG. 6, system 516 may include a network interface 520 (of FIG. 5)implemented as a set of HTTP application servers 600, an applicationplatform 518, tenant data storage 522, and system data storage 524. Alsoshown is system process space 702, including individual tenant processspaces 604 and a tenant management process space 610. Each applicationserver 1000 may be configured to tenant data storage 522 and the tenantdata 523 therein, and system data storage 524 and the system data 525therein to serve requests of user systems 512. The tenant data 523 mightbe divided into individual tenant storage areas 612, which can be eithera physical arrangement and/or a logical arrangement of data. Within eachtenant storage area 612, user storage 614 and application metadata 616might be similarly allocated for each user. For example, a copy of auser's most recently used (MRU) items might be stored to user storage614. Similarly, a copy of MRU items for an entire organization that is atenant might be stored to tenant storage area 612. A UI 530 provides auser interface and an API 532 provides an application programmerinterface to system 516 resident processes to users and/or developers atuser systems 512. The tenant data and the system data may be stored invarious databases, such as one or more Oracle™ databases.

Application platform 518 includes an application setup mechanism 538that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage522 by save routines 536 for execution by subscribers as one or moretenant process spaces 604 managed by tenant management process 610 forexample. Invocations to such applications may be coded using PL/SOQL 534that provides a programming language style interface extension to API532. A detailed description of some PL/SOQL language embodiments isdiscussed in commonly owned co-pending U.S. Provisional PatentApplication 60/828,192 entitled, PROGRAMMING LANGUAGE METHOD AND SYSTEMFOR EXTENDING APIS TO EXECUTE IN CONJUNCTION WITH DATABASE APIS, byCraig Weissman, filed Oct. 4, 2006, which is incorporated in itsentirety herein for all purposes. Invocations to applications may bedetected by one or more system processes, which manages retrievingapplication metadata 616 for the subscriber making the invocation andexecuting the metadata as an application in a virtual machine.

Each application server 600 may be communicably coupled to databasesystems, e.g., having access to system data 525 and tenant data 523, viaa different network connection. For example, one application server 600₁ might be coupled via the network 514 (e.g., the Internet), anotherapplication server 600 _(N-1) might be coupled via a direct networklink, and another application server 600 _(N) might be coupled by yet adifferent network connection. Transfer Control Protocol and InternetProtocol (TCP/IP) are typical protocols for communicating betweenapplication servers 600 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network interconnect used.

In certain embodiments, each application server 600 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 600. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 600 and the user systems 512 to distribute requests to theapplication servers 600. In one embodiment, the load balancer uses aleast connections algorithm to route user requests to the applicationservers 100. Other examples of load balancing algorithms, such as roundrobin and observed response time, also can be used. For example, incertain embodiments, three consecutive requests from the same user couldhit three different application servers 600, and three requests fromdifferent users could hit the same application server 600. In thismanner, system 516 is multi-tenant, wherein system 16 handles storageof, and access to, different objects, data and applications acrossdisparate users and organizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 516 to manage their salesprocess. Thus, a user might maintain contact data, leads data, customerfollow-up data, performance data, goals and progress data, etc., allapplicable to that user's personal sales process (e.g., in tenant datastorage 522). In an example of a MTS arrangement, since all of the dataand the applications to access, view, modify, report, transmit,calculate, etc., can be maintained and accessed by a user system havingnothing more than network access, the user can manage his or her salesefforts and cycles from any of many different user systems. For example,if a salesperson is visiting a customer and the customer has Internetaccess in their lobby, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 516 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS should have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant-specific data, system 516 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 512 (which may be client systems)communicate with application servers 600 to request and updatesystem-level and tenant-level data from system 516 that may requiresending one or more queries to tenant data storage 522 and/or systemdata storage 524. System 516 (e.g., an application server 600 in system516) automatically generates one or more SQL statements (e.g., one ormore SQL queries) that are designed to access the desired information.System data storage 524 may generate query plans to access the requesteddata from the database.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects according to the present invention. It should be understood that“table” and “object” may be used interchangeably herein. Each tablegenerally contains one or more data categories logically arranged ascolumns or fields in a viewable schema. Each row or record of a tablecontains an instance of data for each category defined by the fields.For example, a CRM database may include a table that describes acustomer with fields for basic contact information such as name,address, phone number, fax number, etc. Another table might describe apurchase order, including fields for information such as customer,product, sale price, date, etc. In some multi-tenant database systems,standard entity tables might be provided for use by all tenants. For CRMdatabase applications, such standard entities might include tables forAccount, Contact, Lead, and Opportunity data, each containingpre-defined fields. It should be understood that the word “entity” mayalso be used interchangeably herein with “object” and “table”.

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. U.S. patent application Ser. No.10/817,161, filed Apr. 2, 2004, entitled “Custom Entities and Fields ina Multi-Tenant Database System”, and which is hereby incorporated hereinby reference, teaches systems and methods for creating custom objects aswell as customizing standard objects in a multi-tenant database system.In certain embodiments, for example, all custom entity data rows arestored in a single multi-tenant physical table, which may containmultiple logical tables per organization. It is transparent to customersthat their multiple “tables” are in fact stored in one large table orthat their data may be stored in the same table as the data of othercustomers.

Method for Using the Environment (FIGS. 5 and 6)

FIG. 7 shows a flowchart of an example of a method 700 of usingenvironment 10. In step 710, user system 12 (FIGS. 5 and 6) establishesan account. In step 712, one more tenant process space 104 (FIG. 6) areinitiated on behalf of user system 12, which may also involve settingaside space in tenant space 112 (FIG. 6) and tenant data 114 (FIG. 6)for user system 12. Step 712 may also involve modifying applicationmetadata to accommodate user system 12. In step 714, user system 12uploads data. In step 716, one or more data objects are added to tenantdata 114 where the data uploaded is stored. In step 718, method 4 may beimplemented. In another embodiment, although depicted as distinct stepsin FIG. 4, steps 402-410 may not be distinct steps. In otherembodiments, method 400 may not have all of the above steps and/or mayhave other steps in addition to, or instead of, those listed above. Thesteps of method 400 may be performed in another order. Subsets of thesteps listed above as part of method 400 may be used to form their ownmethod.

Method for Creating the Environment (FIGS. 5 and 6)

FIG. 8 is a method of making environment 10, in step 802, user system 12(FIGS. 5 and 6) is assembled, which may include communicatively couplingone or more processors, one or more memory devices, one or more inputdevices (e.g., one or more mice, keyboards, and/or scanners), one ormore output devices (e.g., one more printers, one or more interfaces tonetworks, and/or one or more monitors) to one another.

In step 804, system 16 (FIGS. 5 and 6) is assembled, which may includecommunicatively coupling one or more processors, one or more memorydevices, one or more input devices (e.g., one or more mice, keyboards,and/or scanners), one or more output devices (e.g., one more printers,one or more interfaces to networks, and/or one or more monitors) to oneanother. Additionally assembling system 16 may include installingapplication platform 18, network interface 20, tenant data storage 22,system data storage 24, system data 25, program code 26, process space28, UI 30, API 32, PL/SOQL 34, save routine 36, application setupmechanism 38, applications servers 100 ₁-100 _(N), system process space102, tenant process spaces 104, tenant management process space 110,tenant space 112, tenant data 114, and application metadata 116 (FIG.6).

In step 806, user system 12 is communicatively coupled to network 104.In step 808, system 16 is communicatively coupled to network 104allowing user system 12 and system 16 to communicate with one another(FIG. 6). In step 810, one or more instructions may be installed insystem 16 (e.g., the instructions may be installed on one or moremachine readable media, such as computer readable media, therein) and/orsystem 16 is otherwise configured for performing the steps of methods3A, 3B and 4. For example, as part of step 810, one or more instructionsmay be entered into the memory of system 16 for scoring and rankingknowledge articles.

In another embodiment, although depicted as distinct steps in FIG. 8,steps 802-810 may not be distinct steps. In other embodiments, method800 may not have all of the above steps and/or may have other steps inaddition to, or instead of, those listed above. The steps of method 800may be performed in another order. Subsets of the steps listed above aspart of method 800 may be used to form their own method.

Extensions and Alternatives

In another embodiment, a different normalization may be used, such asassigning −3 to one star, −2 to two stars, 2 to four stars, and 3 tofive stars. In another embodiment, the user may have fewer or more starsto choose from (zero stars or one star to three stars or ten stars). Inanother embodiment rather than stars another symbol may be used, such ashow many thumbs up and/or down. In an embodiment, there is nonormalization. In another embodiment, the user directly chooses thepoints on scale that starts from a negative number and ends with apositive number. In an embodiment, the user may have multiple systems ofrating an article, and the user may choose to make an entry in each. Forexample, there may be an option to indicate that an article is the“best” article and/or the user may be able to choose to indicate theuser “liked” or “disliked” the article in addition to or instead ofassigning a number of points or stars to the article. An entry in eachsystem of rating the article may affect the score of the article, by adifferent amount.

In an embodiment, prior search results from prior searches may be savedalong with the prior rankings, and when a new search is received withthe same search term, the prior results may be retrieved, and therankings may be updated based on the updates to the score since the lastsearch, and then the search results may be returned to the user witharticles placed in the order of the new ranking. In an embodiment, theamount of time that the user keeps the article open may affect thescore.

In an alternative embodiment, the score may be computed from theformula, s(t_(i))=Δ(t_(i))+s(t_(i-1))/κ, where κ is a positive constantgreater than 1, or using the formulas(t_(i))=Δ(t_(i))+s(t_(i-1))e^(−α(t) ^(i) ^(-t) ^(i-1) ⁾. Using theformula, s(t₁)=Δ(t_(i))+s(t_(i-1))/κ ors(t_(i))=Δ(t_(i))+s(t_(i-1))e^(−α(t) ^(i) ^(-t) ^(i-1) ⁾ may reduce thenumber of times that all the scores need to be divided by a constant toprevent overflow errors or essentially eliminate the need for dividingall of the scores by a constant.

Each embodiment disclosed herein may be used or otherwise combined withany of the other embodiments disclosed. Any element of any embodimentmay be used in any embodiment.

While the invention has been described by way of example and in terms ofthe specific embodiments, it is to be understood that the invention isnot limited to the disclosed embodiments. To the contrary, it isintended to cover various modifications and similar arrangements aswould be apparent to those skilled in the art. Therefore, the scope ofthe appended claims should be accorded the broadest interpretation so asto encompass all such modifications and similar arrangements.

What is claimed is:
 1. A method comprising: determining, by a host system of a database system, the host system having a processor system including one or more processors and a storage system, an increment to a score of an article based on: at least one vote for the article, the at least one vote being weighted higher for votes from external users than votes from internal users, the internal users being internal with respect to an organization controlling the article, the external users being external with respect to the organization, a reference to the article in at least one other article, the reference including a link in the at least one other article, the link providing access to the article, a score of the at least one other article, at least one visitation to the article by at least one user, and a ratio of a number of occurrences of a phrase in the article to a number of words in the article; weighting, by the host system, the increment to the score, so that more recent increments to the score have a higher weight than less recent increments; and storing the score in association with the article in the storage system as an indication of a likelihood that users will be interested in the article. 